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Abstract 


Unmanned  Aerial  Vehicles  (UAVs)  arc  rapidly  becoming  an  integral  part  of  contemporary 
military  operations.  In  light  of  developments  in  UAV  flight  and  payload  technologies,  the  role  of 
UAV  operators  is  evolving  into  that  of  supervisory  controller  of  complex  automated  system. 
However,  it  is  well  established  that  human  operators  perform  poorly  in  this  role.  A  growing  body 
of  literature  points  to  the  crucial  role  that  trust  in  automation  plays  in  determining  the  efficacy  of 
human  monitoring  of  automated  systems.  A  recent  compilation  of  research  on  tmst  in  automation 
suggest  that  providing  operators  with  information  related  to  automation  process  and  context- 
specific  automation  reliability  is  essential  in  promoting  appropriate  tmst  in  automation. 


This  report  presents  a  research  plan  to  create  innovative  design  concepts  for  cognitive  artifacts 
that  communicate  these  two  types  of  information  to  UA  V  operators  for  the  purpose  of 
engendering  appropriate  trust  in  automated  systems.  It  includes  a  categorization  of  the  emerging 
automation  technologies  in  UAV  systems,  an  analysis  of  the  changing  tasks  of  UAV  crews,  and 
a  selective  review  of  relevant  human-automation  research.  The  report  then  introduces  two 
human-machine  interface  concepts  for  selected  types  of  UAV'  automation  and  proposes  a 
research  plan  for  testing  and  evaluating  these  design  concepts. 

Resume 

Lcs  vchicules  acricns  sans  pilotc  (UAV)  sont  rapidement  cn  train  dc  devenir  unc  panic  integrante 
dcs  operations  militaires  contemporaincs.  A  la  lumierc  des  progres  realises  dans  lcs  technologies 
dc  vol  et  dc  charge  utile  des  UAV,  le  role  dcs  operateurs  d’UAV  est  cn  train  dc  sc  transformer  en 
cclui  de  controlcur-superviscur  de  systeme  automatise  complcxc.  Ccpcndant.  ll  est  bien  ctabli 
que  les  operateurs  humains  sont  nial  adaptes  pour  cc  role.  Une  masse  croissante  de 
documentation  temoigne  du  role  crucial  que  jouc  la  confiance  dans  I'automatisation  pour 
determiner  I'cfficacitc  du  controlc  humain  sur  dcs  systemes  automatiscs.  D’apres  une  rccentc 
compilation  dc  la  recherche  mencc  sur  la  confiance  dans  I'automatisation.  ll  scrait  essenticl  de 
procurer  aux  operateurs  des  informations  sur  le  processus  d’automatisation  ct  sur  la  fiabilite  de 
I'automatisation  en  contextc  specifique  pour  promouvoir  un  degre  approprie  dc  confiance  dans 
I'automatisation. 


Le  present  rapport  prcsente  un  plan  de  recherche  visant  a  creer  des  conceptions  novatriccs  pour 
les  artefacts  cognitifs  communiquant  ccs  deux  types  dTnformations  aux  operatcurs  d’UAV  dans 
le  but  d’engendrer  la  confiance  appropricc  dans  lcs  systemes  automatises.  Ccla  inclut  une 
categorisation  des  nouvcllcs  technologies  d’automatisation  emergeant  dans  les  systemes  d'UAV, 
une  analyse  dcs  tachcs  changcantcs  des  equipages  d’UAV',  et  un  examen  sclcctif  de  la  recherche 
pertinente  hommc-automatisation.  Le  rapport  introduit  ensuite  deux  concepts  d’ interface 
hommc-machine  pour  dcs  types  sclcctionncs  d'automatisation  d'UAV  ct  il  propose  un  plan  de 
rccherches  pour  eprouver  ct  evaluer  ccs  conceptions. 


Executive  Summary 

This  article  presents  an  experimental  plan  for  the  evaluation  of  theory -driven  human-automation 
interface  concepts  in  the  domain  of  Unmanned  Acnal  Vehicles  (UAVs).  The  experiment  is 
motivated  by  the  evolution  of  UA V  technology  towards  more  highly  automated  systems  and  the 
relegation  of  human  operators  to  the  role  of  supcrvisoiy  controller.  Humans  have  consistently 
demonstrated  themselves  to  be  ill-suited  for  this  role;  thus,  we  anticipate  that  future  UAV 
systems  will  be  susceptible  to  human-automation  breakdowns  leading  to  mission  failures. 

In  an  attempt  to  head  off  such  failures,  human  factors  researchers  are  exploring  the  role  of 
human  trust  in  automation  as  a  means  of  managing  the  uncertainty  inherent  in  mixed-initiative 
systems.  A  recent  comprehensive  review  of  the  trust  literature  yields  design  guidelines  for 
engendering  appropriate  levels  of  trust  in  automation.  We  followed  these  guidelines  to  produce 
example  interface  artifacts  that  show  information  about  the  automation  process  (i.c.,  the 
automation  algorithm)  and  relevant  context  information  that  affects  the  automation's  behaviour. 

We  propose  an  outline  for  a  simulator  study  to  evaluate  these  interface  concepts  with  human 
operators.  The  study  casts  UAV  operators  opposite  sensor  fusion  automation  that  identifies 
targets  based  on  analysis  of  multiple  sensor  images.  The  task  of  the  joint  human-automation 
system  is  to  correctly  identify  friendly  and  hostile  military  targets  in  a  scene.  The  UAV'  operator* 
will  be  assigned  to  four  groups  to  test  the  differential  effects  of  providing  information  about 
automation  context  and  process.  We  hypothesize  that  providing  cither  or  both  of  these  types  of 
information  about  automation  will  engender  more  appropriate  trust  in  the  automation  and  lead  to 
performance  improvements  over  a  control  condition  lacking  this  information.  The  results  would 
constitute  a  unique  and  substantial  contribution  to  both  the  human-automation  interaction 
literature  and  to  the  ongoing  development  of  UAV  automation. 


Sommaire 


Lc  present  article  presente  un  plan  experimental  pour  1'cvaluation  dcs  concepts  d*  interface 
hommc-automatisation  a  base  theorique  dans  le  domainc  des  vchicules  aeriens  sans  pilote  (UAV). 
L ‘experimentation  cst  motivee  par  le  fait  que  la  technologic  dcs  UAV  cvoluc  vers  dcs  systemes 
plus  hautement  automatiscs  ct  que  les  opera teurs  humains  sont  rclcgucs  au  role  dc  controlcurs- 
superviseurs.  Les  humains  ont  constammcnt  demontre  qu‘ils  sont  mal  adaptes  pi>ur  cc  role;  dcs 
lors.  il  y  a  lieu  dc  croirc  que  les  future  systemes  d‘UAV  seront  sujets  a  des  pannes  d’ interface 
hommc-automatisation  entrainant  dcs  cchccs  de  missions. 


Pour  tenter  d'eviter  de  tels  cchccs.  les  chcrchcure  specialises  dans  les  factcurs  humains  cxplorent 
actucllcmcnt  le  role  de  la  confiancc  humainc  dans  1‘automatisation  comme  moycn  dc  gerer 
l‘ incertitude  mhcrente  aux  systemes  a  initiative  mixte.  Un  examcn  exhaustif  mcnc  rccemmcnt 
sur  la  documentation  relative  a  la  confiancc  a  fait  rcssortir  les  lignes  directrices  de  conception 
pouvant  engendrer  des  niveaux  appropnes  dc  confiancc  dans  1‘automatisation.  Nous  avons  suivi 
ccs  lignes  directrices  pour  produirc  dcs  exemples  d ‘artefacts  d‘ interface  qui  niontrcnt 
l‘ information  sur  le  processus  d’automatisation  (par  excmple.  1‘algonthme  d’automatisation)  ct 
1‘information  sur  le  contextc  pertinent  qui  affecte  le  comportement  dc  I’automatisation. 


Nous  proposons  un  apcriju  d'une  etude  par  simulatcur  visant  a  cvaluer  ccs  concepts  d* interface 
avee  dcs  operatcurs  humains.  L'etude  fait  entrer  en  jeu  les  operatcurs  d*UAV  cn  les  mettant  en 
opposition  a  l‘automatisation  avee  fusion  dcs  capteurs  qui  idcntiiie  les  ciblcs  a  panir  de  1‘analysc 
dcs  images  dc  capteurs  multiples.  La  tachc  du  systeme  conjoint  hommc-automatisation  consistc  a 
identifier  corrcctement  les  cibles  militaircs  amics  ct  hostiles  dans  une  scene  donnec.  Les 
operatcurs  UAV  seront  assignes  a  quatre  groupcs  charges  d'eprouver  les  effets  diffcrenticls 
rattachcs  au  fait  de  fournir  de  l‘ information  sur  le  contextc  dc  rautomatisation  ct  sur  le  processus 
d'automatisation.  Nous  formulons  l‘hypothcsc  que  le  fait  dc  fournir  l*un  ou  Pautre  dc  ces  tN'pcs 
d’ informations,  ou  les  deux,  au  sujet  de  1‘automatisation  suscitcra  une  confiancc  plus  appropnee 
dans  I'automatisation  ct  de  mcillcurcs  performances  que  dans  une  situation  dc  controlc  ou  ccttc 
information  cst  absente.  Les  rcsultats  constitueraient  une  contribution  unique  ct  substantielle 
autant  a  la  documentation  de  I ‘interaction  hommc-automatisation  qu'au  devcloppcmcnt  continu 
dc  rautomatisation  des  UAV. 


Introduction 

Unmanned  Aerial  Vehicles  (UAVs)  arc  rapidly  becoming  an  integral  part  of  military  operations, 
serving  mission-critical  roles  previously  allocated  to  vehicles  operated  by  humans  (21].  Many  of 
these  missions  arc  of  the  “dull,  dirty  and  dangerous’*  variety1 2  ( l  ]  where  supplanting  a  human 
operator  with  a  machine  is  welcome  on  many  levels.  However,  anticipated  developments  in 
UAV  flight  and  payload  technologies  suggest  that  the  role  of  the  UAV  operator  is  evolving  into 
that  of  a  supervisory  controller  of  a  complex  system  (9],  a  task  for  which  the  human  is  ill-suited. 

Two  characteristics  of  complex  systems  that  challenge  human  operators  are  uncertainty  and 
automation*.  Uncertainty  refers  to  the  inevitable  incompleteness  of  sensed  data.  This  could  be  a 
function  of  an  absence  of  sensors,  limitations  in  sensor  fidelity,  impairment  by  environmental 
conditions,  or  degradation  in  sensor  performance  over  time.  The  result  is  that  the  human  operator 
can  never  know  the  true  state  of  the  world,  and  her  impoverished  view  of  the  world  will  rarely 
match  that  of  any  other  actor.  Automation,  in  effect,  serves  as  an  additional  actor  in  complex 
systems.  To  varying  degrees  it  acquires,  analyses,  makes  decisions  about,  and  acts  on  its  own 
impoverished  set  of  sensed  data.  One  of  the  many  extraordinary  challenges  of  human  interaction 
with  complex  systems  is  supporting  human-automation  interaction  in  the  face  of  uncertainty. 

A  key  insight  into  addressing  this  challenge  is  that  human  reactions  to  computers  under 
uncertainty  parallel  interpersonal  reactions  under  similar  conditions.  Trust  is  the  attitude  that 
another  agent  (be  it  human  or  machine)  will  act  to  achieve  a  person’s  goals  in  uncertain 
situations  [24].  This  attitude  plays  a  major  role  in  guiding  the  decision  to  rely  on  that  agent.  For 
this  reason,  human-automation  trust  has  emerged  as  a  central  human  factors  research  theme.  And 
because  human-automation  interaction  occurs  through  cognitive  artifacts  |c.g..  training, 
procedures,  decision  support  systems,  human-machine  interfaces),  it  is  imperative  that  these 
artifacts  be  designed  to  engender  appropriate  trust  in  automation  under  uncertainty. 

While  the  design  goal  is  clear,  there  is  very  little  consensus  in  the  human  factors  community 
regarding  which  cognitive  artifacts  hold  the  most  promise  for  supporting  human  operators 
performing  supervisory  control.  Moreover,  these  are  competing  approaches  designing  these 
artifacts.  For  example,  various  researchers  have  proposed  design  guidelines  for  human- 
automation  interfaces  (7|(24).  To  date,  there  has  been  no  direct  empirical  study  of  any  of  these 
sets  of  guidelines'. 

We  thus  observe  in  the  UAV  domain  a  confluence  of  changing  roles  for  human  operators  and  a 
scarcity  of  disciplinary  understanding  of  how  to  support  humans  in  that  role.  Specifically,  we 
anticipate  that  UAV  operator*  will  increasingly  be  faced  with  the  challenge  of  supervising 
complex  automation  that  is  susceptible  to  uncertainty  and  we  recognize  that  empirical  support 


1  "Dull*  referring  to  cxtcn&d  mission*  with  low  event  rate*  te.g..  patrolling  a  'no-fly*  zoncl:  "dirty*  referring  lo  mission* 
undertaken  in  cnvironnxntt  that  are  no!  conducive  to  healthy  or  safe  operation  (c.g..  chemical  and  biological  agent  sensing!: 
"dangerous"  referring  to  mission*  that  expose  aircraft  and  aircrews  to  hostile  enemy  action  le  g.,  air  deforce  suppression) 

2  Vicente  (41 1  lirts  eleven  characteristics  of  convex  systems:  Large  problem  space*,  social  dynamics,  heterogeneous 
perspectives,  spatial,  distribution,  dynamic  system  response.  hazard,  coupling  between  subsystem*,  automation,  uncertainty, 
mediated  interaction,  and  disturbances.  Clearly  marc  than  uncertainty  and  automation  arc  present  m  UAV  operations.  We  focus 
on  these  mo  to  narrow  exi  problem. 

1  This  is  not  to  say  that  there  have  been  no  empirical  evaluations  of  foiman*  automation  intcrtxc  concepts  Rather,  none  of  the 
interface  concepts  evaluated  appear  to  hive  been  the  direct  product  of  theory  driven  design  guidelines 


for  theory-driven  design  guidance  is  effectively  absent.  The  goal  of  the  proposed  research  is  to 
apply  Lee  and  See's  |24]  guidelines  for  the  design  of  information  about  automation  for  the 
purpose  of  engendering  appropriate  trust  in  that  automation.  Towards  that  end,  this  report: 

1 .  Discusses  the  changing  tasks  of  UA V  crews, 

2.  Categorizes  UA  V  automation  and  some  of  the  ways  that  uncertainty  manifests  therein, 

3.  Reviews  contemporary  UAV  human-automation  research, 

4.  Introduces  two  human-machine  interface  concepts  for  selected  types  of  UAV  automation, 
and 

5.  Proposes  a  research  plan  for  testing  and  evaluating  those  design  concepts. 

UAV  Work  Domain 

UAV  Sensor  Payload 

Table  1  summarizes  some  characteristics  of  the  two  most  prevalent  sensors  used  in  current  UAV 
systems.  Although  other  sensors  are  in  use  and  more  are  forthcoming,  these  will  suffice  for  our 
current  purposes. 

Table  1.  Summary  of  Sensor  Characteristics 

Characteristics 

An  electro-optic*  infra-red  (EO/IR)  camera  (6)  provides  both  optical  and  infrared 
photographic  imagery.  Digital  enhancement  enables  further  sharpening  of 
images.  IR  sensors  can  detect  heat  sources  but  cannot  spot  vehicles  or  aircraft  on 
the  ground  once  their  engines  are  cold.  They  arc  also  unable  to  penetrate  clouds 
and  darkness  and  are  only  slightly  less  likely  to  be  fooled  by  camouflage.  IR 
sensors  can  also  be  fooled  by  dummy  heat  sources  and  can  be  blocked  to  some 
degree  by  special  IR-nctting.  In  addition  the  UAV  camera  output  is  not 
appropriate  for  wide  area  search  and  monitoring  because  of  its  narrow  field  of 
view. 

Synthetic  Aperture  Radar  (SAR)  uses  microwave  signals  to  provide  all  weather, 
day/night  imagery  of  terrain  features  and  man-made  objects.  The  sensor  also 
provides  range  and  bearing  information.  However,  the  resolution  of  SAR  imagery 
is  inferior  to  that  of  EO/IR  and  requires  a  higher  level  of  skill  to  analyze.  SAR 
images  are  also  subject  to  noise  caused  by  unfavorable  conditions  such  as  rough 
seas  or  large,  metallic  surfaces.  In  addition,  the  radar  system  is  also  susceptible  to 
jamming. 

One  drawback  of  SAR  is  that  it  only  shows  the  position  of  stationary  targets. 
Moving  Target  Indicator  (MTI),  in  contrast,  uses  radar  to  show  only  targets  that 
arc  in  motion. 

Crew  Complement  and  Task  Allocation 

When  we  refer  to  a  UAV.  we  arc  referring  to  a  system  1 1 ).  A  fully  operational  UAV  system 
consists  of  one  or  more  aircraft,  a  payload  of  sensors  and  or  effectors,  a  control  station,  and  a 
data  communications  architecture.  Also  included  in  that  system  is  a  crew  of  human  operators 
with  various  roles  depending  on  the  UAV  system,  mission,  nationality,  and  service  branch. 


Sensor 

EOIR 


SAR.  MTI 


A  US  Defense  Department  road  map  for  UAVs  (30)  lists  the  training  programs  for  several  UAV 
systems,  thereby  identifying  crew  complement  and  roles  for  each  system.  Although  these  differ 
across  platform,  it  is  sufficient  for  our  purposes  to  characterize  common  features  of  UAV 
crewing. 

Generally  speaking,  a  contemporary  UAV  crew  consists  of  a  pilot  (also  known  as  an  air  vehicle 
operator)  and  one  or  more  sensor  (or  payload)  operators  and  a  mission  planner  or  tactical 
navigator  (23)  {39],  The  pilot  is  responsible  for  Hying  the  UAV,  sometimes  using  traditional 
stick  and  rudder  controls  and  at  other  times  by  engaging  an  autopilot  or  waypoint  fixing  (23]  (30] 
The  sensor  operators  arc  responsible  for  manipulating  sensors,  optimizing  them  for  data 
collection,  and  interpreting  the  data  collected  [39].  For  example,  an  operator  might  select  and 
tunc  a  sensor  to  locate  a  target  through  cloud  cover,  identify  the  target,  and  communicate  its 
location  and  identity  to  other  assets  in  the  battlespace.  The  tactical  navigator  acts  as  the  mission 
commander,  and  is  responsible  for  managing  the  use  of  available  resources  (including  multiple 
UAVs  and  the  control  station  itself)  to  accomplish  the  mission  goals  (39]. 

Although  human  factors  research  in  UAV  operation  is  beginning  to  pick  up  speed,  there  arc  few 
publicly  available  analyses  of  UAV  operator  tasks.  Perhaps  the  best  source  is  a  cognitive  task 
analysis  performed  by  Gugerty  ct  al  (1999)  based  on  interviews  with  UAV  crews.  They  present  a 
goal  hierarchy  for  a  typical  Predator  mission  (see  Figure  1 ).  The  analysis  shows  the  emphasis  on 
reconnaissance  missions  and  the  particular  difficulties  involved  in  re-planning  missions. 
However,  there  is  little  treatment  of  automation  in  the  analysis,  perhaps  because,  at  the  time  that 
it  was  completed,  few  of  the  crew  tasks  were  automated. 
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Figure  I.  (kial  hierarchy  for  a  Predator  raUftion. 


A  clearly  defined  future  goal  for  UAV  operations  is  to  reduce  the  ratio  of  opcrators-to-aircraft  by 
having  operators  sen  e  multiple  roles  |45).  The  anticipated  means  of  achieving  this  is  through 
automation.  The  first  role  that  is  likely  to  be  assigned  to  automation  is  flight  control.  Operators 
of  future  UAV  systems  are  envisioned  to  issue  navigation  waypoints  with  flight  paths  planned 
and  executed  by  automation.  Tasks  that  are  currently  allocated  to  the  sensor  operator  and  tactical 
navigator  will  also  become  more  automated.  Automated  target  recognition  and  data  fusion 
algorithms  are  expected  to  relieve  the  sensor  operator  of  much  of  the  information  analysis  burden, 
presenting  him  with  conclusions  to  verify  and  courses  of  action  to  approve. 

Unfortunately,  we  have  not  identified  any  task  analyses  of  operations  of  more  highly  automated 
UA  Vs.  Moreover,  we  have  not  had  direct  access  to  any  subject  matter  experts  to  conduct  our 
own  analyses.  Therefore,  many  of  our  assumptions  about  the  future  role  of  UAV  operators  are 
drawn  from  sources  that  promote  UAV  technology  without  direct  consideration  for  human 
factors  issues.  The  clear  implication  of  these  sources  is  that  the  task  of  the  future  UAV  payload 
operator  in  particular  will  be  increasingly  one  of  monitoring  automation  processes. 

This  foresight  should  raise  warning  flags  with  human  factors  engineers.  Elsewhere  in  aviation  (3) 
and  in  other  domains  (47),  advances  in  automation  technology  have  frequently  failed  to  deliver 
the  expected  improvements  in  system  performance.  These  failures  are  often  attributed  to  poorly 
designed  human-automation  interaction.  There  is  a  sense  in  that  human  factors  community  that 
UAVs  may  be  a  domain  where  we  can  make  an  impact  on  the  design  of  future  UAV  systems  to 
head  off  human-automation  interaction  problems. 

Automation  Categories 

Emerging  automation  technologies  in  UAV  systems  fall  into  several  categories  (40]: 

1 .  Sensor  Fusion:  integrating  information  from  various  sensors. 

2.  C  ommunications:  handling  communication  and  coordination  between  multiple  agents  in 
the  presence  of  incomplete  and  imperfect  information. 

3.  Flight  Path  Planning  and  Trajectory  Generation:  determining  an  optimal  path  and 
enabling  control  maneuvers  to  allow  a  vehicle  to  meet  objectives  and  respect  constraints 
such  as  no-fly  zones. 

4.  Task  Allocation  and  Scheduling:  determining  the  optimal  distribution  of  tasks  amongst  a 
group  of  agents,  with  tune  and  equipment  constraints. 

5.  Cooperative  Tactics:  formulating  an  optimal  sequence  and  spatial  distribution  of 
activities  between  agents  in  order  to  maximize  chance  of  success  in  any  given  mission 
scenario. 

In  many  situations,  these  types  of  automation  will  interact.  For  example,  motion  planning  and 
cooperative  tactics  automation  will  determine  the  path  of  the  UAV.  and  therefore  influence  the 
information  acquired  from  sensors. 

Selection  of  Sensor  Fusion  Automation 

The  current  research  goal  is  to  develop  human-automation  interfaces  that  assist  UAV'  operators 
supervising  less  than  perfectly  reliable  automation  under  uncertainty.  We  are  particularly 
interested  in  howr  environmental  context  may  influence  the  automation  performance  in  specific 
situations  [2]  (7]  (28].  Sensor  fusion  automation  appears  to  fit  our  requirements: 


•  The  output  of  the  sensor  fusion  automation  typically  includes  a  degree  of  uncertainty 
[46).  providing  a  list  of  possible  target  types  and  their  probabilities. 

•  Sensor  fusion  automation  is  not  perfectly  reliable.  Efforts  to  improve  sensor  fusion 
algorithms  arc  underway  [  19],  [29)  and  [46).  but  the  technology  is  still  relatively 
immature. 

•  Sensor  fusion  automation  is  limited  by  incomplete  and  noisy  raw  data  collected  by  the 
sensors,  often  as  a  result  of  environment  conditions.  For  example,  cloud  cover,  dust  and 
smoke  interfere  with  many  sensors.  As  well,  the  data  link  may  be  lost  due  to  obstacles 
between  the  UAV  and  the  control  station  [39).  Finally,  adversaries  may  employ 
countermeasures  to  degrade  the  quality  of  data  collected  [29). 

Sensor  Fusion  Algorithm  Example 

Sensor  fusion  automation  aggregates  data  from  multiple  sensors  to  support  target  detection, 
target  identification  and  tactical  decision-making.  An  application  of  Dempster-Shafer  theory 
provides  a  good  example  of  sensor  fusion  automation  [46].  The  inputs  to  this  sensor  fusion 
automation  arc  two  lists  of  possible  target  types  and  possibilities  provided  by  independent 
automatic  target  recognition  ( ATR)  systems.  Each  ATR  system  processes  a  sensor  image  and 
passes  its  assessment  of  targets  in  the  scene  to  the  sensor  fusion  automation1. 

The  fusion  algorithm  is  built  on  Dempster-Shafer  theory:  a  mathematical  theory  of  evidence  used 
to  combine  independent  pieces  of  information  (or  evidence)  to  find  the  probability  of  a 
hypothesis.  Some  basic  concepts  in  Dempster-Shafer  theory  follow'.  A  frame  of  discernment  (0) 
is  defined  as  the  universal  set  of  all  propositions  being  considered.  For  example,  consider  the 
ease  of  identifying  whether  an  object  is  a  T34  tank.  The  frame  of  discernment  contains  the 
following  propositions: 

•  {T34 1 :  the  object  is  a  T34  tank 

•  !  “T34 ! :  the  object  is  not  a  T34  tank 

•  |T34.^T34J:  it  is  inconclusive  whether  or  not  the  object  is  a  T34  tank  (this  is  called  the 
“ignorance  set*') 

Each  proposition  in  this  set  can  be  assigned  a  degree  of  belie  f  { the  function  Bel)  based  on  the 
reliability  of  sensor  data.  As  an  example,  suppose  sensor  data  determines  an  object  to  be  a  T34 
tank.  The  probability  that  the  sensor  is  reliable  is  0.S  and  the  probability  that  it  is  unreliable  is  0.2. 
These  probabilities  justify  a  degree  of  belief  of  0.8  that  the  object  is  a  T34.  However,  the  degree 
of  belief  that  the  object  is  not  a  T34  is  0.  which  means  that  no  evidence  has  led  to  the  possibility 
that  the  object  is  not  a  T34.  Intuitively,  this  makes  sense  because  the  detection  algorithm  docs 
not  explicitly  test  whether  an  object  fails  to  be  a  T34.  Note  that  a  zero  degree  of  belief  is 
different  from  a  zero  probability,  which  would  mean  that  there  is  no  chance  that  the  object  is  a 
T34.  Mathematically,  the  degree  of  belief  in  this  example  can  be  represented  by: 

•  Bcl(!T34])  =  0.S 

•  Bcl({-T34})  =  0 

•  Bcl({T34,“T34j>  =  0.2 

This  simple  example  illustrates  how'  the  degree  of  belief  for  one  hypothesis  (“Is  the  object  a  T34 
tank?")  can  be  obtained  using  one  item  of  evidence.  Obtaining  the  belief  function  from  multiple 

4  As  ihc  name  implies.  ATR  is  jutonutum  itself  so  this  example  is  coc  of  coupled  nut  coition 


sets  of  uncertain  evidence  (e.g.,  acquired  through  multiple  sensors  on  the  same  UAV,  multiple 
sensors  on  different  UAVs.  or  multiple  instances  from  the  same  sensor  at  multiple  times)  is  more 
complicated. 

For  example,  assume  that  UAV  sensors  on  the  battlefield  scan  an  area  of  terrain  and  output  a  list 
of  candidate  identities  for  each  target  (T34  tank.  M12  tank,  etc.)  with  their  respective  confidence 
levels.  The  identity  w  ith  the  highest  confidence  level  initially  determines  the  active  frame  of 
discernment,  although  this  may  change  once  additional  evidence  is  added.  These  frames  of 
discernment  can  disagree:  one  sensor  may  indicate  that  the  target  is  most  likely  a  T34  tank,  while 
another  may  indicate  that  it  is  probably  an  Ml 2  tank.  Multiple  candidate  lists  can  be  “fused"  to 
produce  an  aggregate  belief  function  for  all  potential  target  identities.  To  do  this,  a  confusion  set 
is  formed  for  each  list  by  taking  all  members  above  a  confidence  level  threshold  o.  The  belief 
function  for  a  frame  of  discernment 

Any  two  confusion  sets  Si  and  S:  are  fused  according  to  one  of  three  scenarios: 

1 .  SI  and  S2  have  the  same  active  frame  of  discernment  (i.c.  the  identity  with  the  highest 
confidence  level  in  each  set  is  the  same).  The  belief  functions  of  the  two  sets  arc  fused 
according  to  Dempster's  rule  of  combination  (  not  described  here). 

2.  SI  and  S2  have  different  frames  of  discernment,  with  some  shared  members.  The  single 
member  w  ith  the  highest  confidence  level  in  cither  set  is  chosen  to  be  the  active  frame  of 
discernment  (say  this  is  II  in  SI).  If  1 1  is  not  in  the  other  list,  we  use  the  belief  function 
of  S 1  as  the  result.  Otherwise,  we  combine  the  belief  functions  of  both  sets  for  1 1 . 

3.  SI  and  S2  have  ditYcrent  frames  of  discernment,  with  no  shared  members.  No  combining 
occurs,  and  we  use  the  belief  function  for  S2. 

The  general  idea  in  the  algorithm  presented  by  Yu.  et  al.  (2004)  is  to  obtain  belief  functions  from 
the  confidence  levels  outputted  by  ATRs.  The  belief  functions  can  then  be  fused  and  refined  as 
additional  evidence  is  collected. 

Analytical  Redundancy 

The  principle  of  analytical  redundancy  suggests  that  providing  information  on  the  context  of  the 
automation  in  the  interface  can  allow  an  operator  to  more  easily  identify  the  exact  source  of 
erroneous  or  unreliable  automation.  An  interface  in  the  U  A  V  sensor  domain  can  incorporate 
analytical  redundancy  by  displaying  the  signals  from  all  elements  in  the  system,  as  shown  in 
Figure  2.  The  output  of  the  UAV  sensor  is  an  image  of  the  field.  This  data  is  processed  by  an 
Automatic  Target  Recognition  (ATR)  system,  which  outputs  a  list  of  candidate  identities  with 
respective  confidence  levels.  In  the  information  aggregation  stage,  confusion  sets  arc  determined 
based  on  a  threshold  o  and  belief  functions  are  calculated.  These  sets  arc  fused  and  the  aggregate 
belief  function  is  outputted. 
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Figure  2:  Cenerir  t  AV  umur  fusion  uuiuiti jImui. 

Human-Automation  Interface  Concept  1 

I  he  display  shown  in  Figure  1  supports  analytical  redundancy  by  revealing  information  about 
the  input  and  output  signals  to  every  clement  in  the  U AV  sensor  system.  Moreover,  it  provides 
the  operator  with  a  description  of  how  the  automation  performs  information  aggregation  by 
displaying  information  at  the  intermediate  steps  of  the  belief  function,  the  confusion  sets  and 
fusion  of  the  sets.  We  believe  that  this  satisfies  the  design  guidelines  for  process  information  as 
described  by  Lee  &  Sec  (2004).  I  he  theory-driven  design  framework  predicts  that  providing  this 
information  will  help  the  operator  to  isolate  any  errors  in  the  system  and  establish  an  appropriate 
level  of  trust  in  the  automation. 
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Klltu re 3:  Cognitive  Artifact  Concept  #1  for  information  aggregation  tinted  un  Deraptter-Sliafer  theory 

I  he  main  elements  in  this  interface  concept  arc  sensors  images  (top),  candidate  list  graph 
(bottom  left),  and  active  candidate  graph  (bottom  right).  7hc  sensor  images  arc  the  output  signals 
from  the  UAV  sensors  (two  arc  assumed  in  this  case),  which  show  the  ground  area  hypothesized 
to  contain  a  tank. 


The  candidate  list  graph  contains  information  on  each  sensor  ATR's  candidate  list  with 
respective  confidence  levels  represented  by  the  height  of  each  non-black  bar.  In  this  example,  the 
stacked  columns  on  the  left  corresponds  to  sensor  I  and  the  columns  on  the  right  corresponds  to 
sensor  2.  Confusion  sets  are  determined  by  applying  a  threshold  o  =  .15  (indicated  by  the  dotted 
line).  Members  which  are  not  pari  of  any  confusion  set  arc  white  to  reduce  their  visual  presence. 
The  belief  function  is  calculated  for  each  member  contained  in  the  confusion  sets  and 
displayed  in  the  following  manner: 

•  Bcl({T} )  is  coloured  orange  if  it  is  the  active  frame  of  discernment,  and  grey  otherwise 

•  Bcl({-,T})  is  coloured  black 

•  Bel({TrT | )  is  not  shown  (its  value  corresponds  to  the  “missing"  space) 

The  active  candidate  graph  provides  a  visualization  of  the  aggregate  belief  function  after  fusion 
of  the  confusion  sets,  and  corresponds  to  the  active  frame  of  discernment.  It  follows  the  same 
colour  scheme  as  the  candidate  list  graph,  such  that  the  operator  can  easily  identify  the  active 
frame  of  discernment  in  both  graphs. 

An  example  where  this  interface  concept  can  be  applied  is  for  a  UAV  that  has  one  SAR  sensor 
for  stationary  object  identification  and  one  MTI  sensor  for  moving  object  identification.  Both 
sets  of  sensor  data  generate  independent  candidate  lists,  but  one  may  be  more  reliable  than  the 
other  under  certain  conditions  (e.g.,  climate.  UAV  altitude,  time  of  day.  prior  knowledge  that  the 
object  is  stationary  or  moving).  The  operator  can  isolate  one  sensor’s  belief  function  in  cases 
where  the  aggregate  belief  function  is  likely  to  provide  misleading  results. 

Human-Automation  Interface  Concept  2 

The  second  interface  concept,  shown  in  Figure  4.  also  incorporates  analytical  redundancy  by 
revealing  information  about  all  signals  in  the  UAV  sensor  system  and  by  displaying  information 
at  the  intermediate  steps  of  the  automation. 

The  main  elements  in  this  interface  concept  arc  sensor  images  (top)  and  a  candidate  list  gauge 
(bottom).  Information  from  Concept  #1  ’s  candidate  list  graph  and  active  candidate  graph  are 
incorporated  into  a  single  interface  element,  the  candidate  list  gauge.  As  in  Concept  #1,  the 
sensor  images  arc  the  output  signals  from  two  UAV  sensors,  showing  the  ground  area 
hypothesized  to  contain  a  tank. 

The  candidate  list  gauge  contains  information  on  each  sensor’s  candidate  list  with  respective 
confidence  levels  represented  by  triangular  markers  pointed  toward  a  value  on  the  gauge  line. 
Sensor  I  data  corresponds  to  orange  markers  with  the  number  “lM  while  sensor  2  data 
corresponds  to  green  markers  with  the  number  “2.”  As  in  Concept  #1.  confusion  sets  are 
determined  by  applying  a  threshold  a  =  .15  (indicated  by  the  dotted  line  in  the  upper  section  of 
the  gauge).  The  gauge  is  vertically  split  in  the  middle,  with  values  for  Bcl({TJ )  above  the  zero 
line  and  values  for  Bel( { — T } )  below  the  zero  line  (for  each  candidate  T).  Note  that  Bcl(  J ) 
values  are  only  computed  for  candidates  belonging  to  confusion  sets.  Bcl(  JTr'T})  is  not  shown, 
although  its  value  can  be  inferred. 
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Hgur6  4:  Cognitive  Artifact  Concept  H2  far  inlw  uuiimi  aggregation  baud  on  Diinpttcr-Shafcr  theory 

llic  candidate  list  gauge  provides  a  visualization  of  the  aggregate  belief  function  for  the  active 
frame  of  discernment,  the  value  of  which  is  represented  by  the  position  of  a  circle.  The  circle 
interface  element  also  indicates  how  the  confusion  set  information  is  fused  os  shown  in  Figure  5. 
O  iCisor  1  *fu2  sen $oi  1 
©  onvscit^l  btffcffurrtcouscd 
O  3*V^^bdtffurtter>us«J 

Figure  5:  Fusion  of  con Tu vi on  Ml  information  indicated  by  the  circle  interface  elrmrnt 

As  in  Concept  ^1,  this  interface  concept  allows  the  operator  to  distinguish  the  confidence  levels 
obtained  from  each  sensor,  by  looking  at  cither  colour  of  the  triangular  marker  or  the  number 
beside  it.  I  his  separation  is  of  value  when  one  sensor  is  known  to  be  more  reliable  under  certain 
environmental  conditions.  The  interface  also  allows  the  operator  to  easily  identify  the  values  for 
Bel(  { T ! )  and  Bcl(  above  and  below  the  zero  line  respectively.  Presenting  the  aggregate 

belief  function  in  the  gauge  allows  the  operator  to  directly  compare  the  component  effects  of 
each  sensor's  belief  function. 


Comparison  of  Human-Automation  Interface  Concepts 

The  two  concepts  presented  here  arc  initial  low-fidelity  prototypes  that  have  nol  undergone  any 
evaluation.  Before  they  arc  included  in  a  research  study,  they  should  be  subjected  to  both 


feasibility  and  usability  analyses.  Such  analyses  arc  included  in  a  research  plan  below,  but  will 
not  be  detailed  here. 

Current  Research  in  UAV  Automation 

UAVs  are  quickly  becoming  the  domain  de  jure  for  human  factors  research  for  at  least  four 
reasons.  First,  the  wide  range  of  UAV  operator  tasks  allows  for  many  types  of  behavioral 
research.  Second,  the  applicability  of  UAVs  across  a  broad  range  of  military  (lethal  and  non- 
lethal)  and  quasi-military  (c.g.,  intelligence  gathering,  border  security)  operations  aligns  with  a 
similarly  broad  range  of  government  funding  sources  for  research  and  development.  Third,  cases 
for  civilian  application  (c.g.,  law'  enforcement,  forest-fire  fighting)  are  plausible  enough  to 
classify  the  technologies  as  dual-use.  Finally,  UAVs  offer  the  allure  of  aviation  without  the 
required  domain  expertise  that  comes  with  piloting. 

For  these  reasons,  a  host  of  UAV  research  programs  are  emerging  to  explore  a  range  of  research 
issues.  This  trend  suggests  two  likely  benefits  for  near-term  human  factors  research.  First,  we 
can  be  optimistic  that  a  well-conceived  research  program  couched  in  the  domain  has  a  reasonable 
chance  of  securing  sustaining  funding  .  Second,  we  can  be  optimistic  that  such  a  program  will 
also  find  outlets  for  its  results  in  quality  industry  and  academic  journals. 

Synthetic  Environments  and  Tasks  in  UAV  Research 

In  this  section,  we  review  the  experimental  environments  currently  in  use  in  IJAV  human  factors 
research.  The  review  for  each  research  programme  includes: 

•  a  description  of  the  synthetic  task  environment, 

•  an  account  of  the  tasks  allocated  to  the  human  operators, 

•  a  characterization  of  the  type  of  automation  employed 

This  review  contributes  to  the  current  work  in  two  ways.  First,  it  will  establish  requirements  for 
a  ‘state  of  practice*  synthetic  environment  that  can  provide  valid  findings  regarding  the 
theoretical  issues  to  be  investigated.  Second,  the  review  will  help  to  identify  the  tasks  of  UAV 
crews  that  we  may  wish  to  emulate.  Detailed  descriptions  of  such  tasks  are  scarce  in  the 
available  literature  and  we  have  not  been  afforded  the  opportunity  to  conduct  a  task  analysis  with 
subject  matter  experts.  The  synthetic  tasks  used  in  the  reference  literature,  while  artificial,  are 
systematically  abstracted  from  corresponding  real-world  tasks  by  other  human  factors 
practitioners  [23).  Therefore,  we  have  some  confidence  that  this  literature  can  help  to  overcome 
our  knowledge  deficiency  with  respect  to  real-world  UAV  operator  tasks. 

Dixon,  Wickens,  et  al. 

Institute:  University  of  Illinois  Institute  of  Aviation.  Aviation  Human  Factors  Division 
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Assuming  th»  the  program  is  earned  out  in  a  jurcdictkm  cligibk  to  access  those  funding 


Synthetic  Environment 

A  series  of  studies  on  human  interaction  with  UAV  automation  have  been  conducted  using  a 
UAV  synthetic  task  environment.  VEGA"  was  used  as  the  overall  graphical  scene  controller  [see 
Figure  6).  The  interface  was  displayed  on  a  19-inch  monitor  with  12X0\I024  resolution,  the 
interface  (see  Figure  6)  was  divided  into  four  view  ports  with  four  dedicated  process  views: 


•  3-D  visual  image  display  with  two  modes: 


o  fixed  viewing  angle  and  zoom  in  'tracking*  mode,  and 
o  variable  v  iewing  angle  and  zoom  in  ‘loitering*  mode 


•  2-D  navigational  display 

•  system  gauges  for  monitoring  for  system  failures,  and 

•  message  box  to  receive  instructions 


I  he  control  devices  include  a  digital  3D  joystick  and  an  X-Key  20-button  keypad.  The  joystick  is 
used  to  manipulate  the  UAV  and  the  camera  while  the  keypad  is  used  to  indicate  responses. 


Figure  6.  Hit*  LUX*  *> nt belie  rtprrimenl  tavirunitient  (frnm  |  IS]) 


A  Miltwan.'  application  development  dcMgricd  (or  the  creation  and  deployment  of  real-tune  3D  Mtnutation.  training, 
and  \i.*uali/ation  application*. 


Synthetic  Tasks 

Undergraduate  and  graduate  students  were  asked  to  tly  one  UAV  through  several  mission  legs, 
while  completing  three  mam  tasks: 

1 .  Mission  Completion:  subjects  were  tasked  (via  the  message  box)  with  traveling  to  map 
locations  to  perform  a  target  inspection  and  reporting  task. 

2.  Target  Search:  in  between  map  locations,  subjects  were  tasked  with  detecting  and 
reporting  low  visibility  targets  of  opportunity.  These  targets  were  camouflaged  and  their 
occurrence  and  location  were  unknown  to  participants.  Moreover,  they  were  much 
smaller  than  those  in  the  mission  completion  task. 

3.  System  Monitoring:  when  a  system  gauge  went  “out  of  bounds",  participants  had  to 
press  a  button  to  detect  the  system  failure,  indicate  which  gauge  had  failed,  and  then 
report  the  current  location  of  the  UAV.  The  detection  and  acknowledgement  sub-task 
appears  to  be  taken  from  the  Multi- Attribute  Task  (MAT)  battery. 

The  measurements  taken  in  these  experiments  include  the  objective  performance  in  different 
tasks  as  well  as  some  subjective  ratings,  such  as  tmst.  Trust  was  assessed  by  questionnaire:  for 
example,  “How  do  you  assess  the  trustw  orthiness  of  the  auto-pilot?%*  (( 15),  pp.64) 

Automation 

The  studies  have  examined  several  types  of  pilot  interaction  w  ith  perfect  and  imperfect 
automation.  Each  of  the  three  tasks  above  can  be  assigned  to  the  participant  or  to  automation  aids. 
The  target  detection  automation  was  designed  to  provide  an  auditory  signal  w  hen  it  sensed  a 
target  of  opportunity  in  the  3-D  image  display.  However,  it  did  not  designate  the  specific 
location  of  the  target. 

Ruff,  et  al. 

Institute:  Air  Force  Research  Lab,  Human  Effectiveness  Directorate  &  Sytronics.  Inc. 
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Synthetic  Environment 

The  purpose  of  the  research  is  to  investigate  the  human  factors  in  supervisory  control  of  multiple 
IJAVs.  It  used  the  Multi-Model  Immersive  Intelligent  Interface  for  Remote  Operation  (MIIIRO) 
test  bed.  which  is  described  as  “a  generic  UAV  operator  interface  simulation  testbed'*  (|37|.  pp. 
218).  The  MIIIRO  workspace  consists  of  two  monitors  as  shown  in  Figure  7.  The  tactical 
situation  display  on  the  let!  shows  the  UAV  routes,  suggested  route  re-plans,  waypoints,  targets, 
threats,  and  unidentified  aircraft.  The  image  management  display  on  the  right  show  s  camera 
images  taken  by  UAVs  with  hostile  targets  highlighted  by  an  automatic  target  recognizer  (AT  R). 
Below  the  image  is  an  image  queue.  The  control  devices  arc  a  keyboard  and  a  mouse. 


Figurt!  7.  The  MIIIUO  %s  ntbelk  Utk  environment:  Taiticul  Situation  l)i*plu>  |K*ll >  unit  Ima^i*  Mamie  nuiol 
Dhplay  (rifthl)  (from  |37|> 

Synthetic  tasks 

Participants  (whose  characteristics  were  not  mentioned)  were  required  to  complete  lo  tour 
prioritized  tasks: 

1 .  Respond  to  a  highly  unexpected,  non-routine,  high-priority  event. 

2.  Accept  reject  route  replanning  solutions  suggested  by  automation  in  response  to 
unanticipated  targets  and  threats. 

3.  Accept  reject  automated  target  identifications. 

4.  Acknowledge  changes  in  mission  mode. 

I  he  measurements  taken  from  the  experiments  |37)  include  the  time  and  accuracy  in  each  task, 
subjective  rating  of  task  difficulty,  workload,  trust,  automation  and  self-confidence  The 
approaches  for  soliciting  subjective  rating  were  not  mentioned  in  the  paper. 

Automation 

Two  kinds  of  automation  were  employed,  automatic  route  planning  and  automatic  target 
detection.  For  both  automated  systems,  the  reliability  and  automation  level  (management -by- 
consent  vs.  managcmcnt-by-cxccption)  served  as  independent  variables. 
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Synthetic  Environment 

CERTT*s  UAV-STE  simulates  the  ground  control  operations  of  a  UAV.  The  interface  allows 
users  to  navigate  the  UAV  to  a  position  and  perform  a  reconnaissance  task  of  photographing 
designated  targets.  CERTT  uses  ihe  setup  to  study  team  cognition  with  experiments  involving  a 
three-person  team. 

The  experimental  set-up  is  composed  of  four  participant  consoles  and  one  experimenter  control 
station,  as  shown  in  f  igure  8  and  Figure  9. 


Htturt*  K:  Purliclpjnt  rnntuta 


Hgtiri'  9:  Experimenter  Control  Station 

Each  participant  console  contains  the  following:  two  computers,  one  monitor,  headsets  and  an 
intercom.  Each  team  member  has  a  separate  role,  either  Air  Vehicle  Operator  (A VO),  Payload 
Operator  (PLO).  or  Tactical  Navigator.  Their  respective  displays  are  shown  in  Figures  3-5. 
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ITic  STE  is  Windows-based  and  implemented  through  user-defined  objects  developed  in  the 
Rapid1  M  Development  Environment.  I  he  setup  involves  a  combination  of  custom-built  and  off- 
the-shelf  hardware. 


Synthetic  t  asks 

This  STE  is  used  to  study  UAV  operations  and  team  coordination.  The  individual  responsible 
tasks  for  each  operator  arc  shown  below.  However*  to  fulfill  the  mission,  they  have  to  coordinate 
in  order  to  maneuver  their  UAV. 

•  Air  vehicle  operator:  controls  UAV  airspeed,  heading,  and  altitude  and  monitor  air 
vehicle  systems. 

•  Sensor  operator:  control  camera  settings,  takes  photos,  and  monitors  camera  systems. 

•  Tactical  navigator:  navigator,  mission  planner,  plans  route  from  target  to  target  under 
constraints. 


In  the  experiment.  Cooke,  et  al.  (2000)  recruited  I  OH  New  Mexico  State  University  students  in 
the  experiment.  Outcomes,  team  process  behaviors,  tarn  situation  awareness,  taskwork 
knowledge  and  teamwork  knowledge  were  measured  in  the  experiment.  Outcome  included 
mission  completion  rate,  time  to  complete  mission,  and  experimenter  ratings  of  team 
performance. 

Automation: 

Automation  settings  are  not  mentioned  in  the  papers. 
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Synthetic  Task  Environment 

Rather  than  describing  a  research  program,  this  article  describes  the  development  of  a  UCAV 
synthetic  task  environment  for  human  factors  studies  of  multiple  UAV  control.  This  synthetic 
task  environment  consists  of  interlace  software  developed  in  VEGA  and  simulation  software 
developed  in  JAVA.  A  screenshot  of  the  interface  is  shown  in  Figure  13.  A  status  panel  and 
control  panel  arc  layered  at  the  lower  part  of  the  interlace,  and  a  map  display  is  shown  on  the 
upper  part.  The  map  display  is  further  divided  into  three  different  view  ports:  satellite  view, 
camera  view  and  following  view.  The  interface  is  displayed  on  a  21  inch  monitor,  and  input 
devices  include  a  mouse,  keyboard  and  voice  [35 ]. 


Figure  13.  Senvmhirt  ol  the  human  contrullcr  inft  rlucv  (from  |35|> 


Synthetic  Tasks 

The  interface  and  simulation  design  was  driven  by  a  suppression  of  enemy  aerial  defenses 
mission.  Therefore,  the  synthetic  task  environment  has  mechanism  to  support  various  tasks  in 
control  of  UCAV.  including:  “navigation,  flight  paths.  UCAV  updates,  target  tracking,  target 
identification,  target  destruction,  and  elements  that  aid  the  UCA  Vs"  (f35),  pp.  1036).  In  a  typical 
UCAV  operation  scenario,  the  operator  would  complete  the  following  tasks: 

1 .  System  Monitoring:  monitor  and  control  the  UCA Vs  for  unanticipated  variability. 

2.  Route  Planning:  make  adjustments  to  the  flight  paths  to  detect,  identify  and  destroy 
target  and  return  to  base. 

3.  Target  Detection  &  Identification:  detect  and  identify  targets  via  simulated  sensors. 

4.  Target  Destroy:  select  the  right  kind  of  ammunition  to  engage  different  types  of  target. 

Automation 

Based  on  the  description  provided  in  the  paper,  the  UCAV  waypoint  tracking  and  target 
acquisition  arc  automated.  T  he  target  acquisition  serves  as  an  example  of  sensor  fusion 
automation.  A  simulated  long-range  sensor  detects  targets  while  a  second,  short-range,  sensor 
identifies  the  targets.  No  explicit  information  is  provided  about  the  automation  in  the  interface. 
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Synthetic  Task  Environment 

The  purpose  of  this  research  [5)  is  to  study  how  cueing  data  from  Moving  Target  Indicator  (MTI) 
automation  can  be  communicated  to  a  Predator  l)AV  camera  operator.  The  synthetic  task 
environment  simulates  the  Predator  UAV  user  interface.  The  user  interface  is  displayed  on  two 
monitors,  the  left  one  shows  the  simulated  UAV  video,  the  right  one  shows  a  situational 
awareness  map.  The  cues  generalized  by  the  MTI  algorithm  are  highlighted  on  the  situational 
awareness  map  with  red  dots. 


Hull  re  14.  SernnUurt  tif  the  *>nflictic  lank  environment:  Simulated  LAV  Video  (Lett).  M II  Cue 
Silualiunal  A^litaeu  DivpUv  (riuiit)  (quoted  from  |5|> 

Synthetic  Tasks 

The  task  designed  for  future  experiments  is  to  collect  battlefield  convoy  information.  This  task 
involves  selecting  a  cue.  directing  the  UAV  video  sensor  to  collect  the  information  from  the  cued 
location,  searching  tor  and  identifying  the  convoys,  counting  the  number  of  vehicles  in  convoys 
and  acquiring  an  image  of  them  [5). 

Automation 

Ihe  automation  algorithm  studied  in  this  research  utilizes  the  information  from  MTI  Radar  to 
predict  the  location  and  activity  of  convoys. 

Summary  of  Findings  Regarding  UAV  Research 

1 .  The  synthetic  task  environments  employed  in  current  human  factors  UAV  research 
domain  arc  generally  high  in  fidelity.  They  offer  realistic  images  from  payload  sensors 
and  simulate  various  tasks  in  UAV  operation  |5|  (14]  [IS]  [16|  1 I7|[43| .  A  high  fidelity 
synthetic  task  environment  increases  the  likelihood  that  a  controlled  experiment  would 
provide  results  that  arc  relevant  to  the  real  world.  However,  a  low  fidelity  synthetic  task 
environment  may  suffice  for  some  research  [27 1  which  is  only  interested  in  one  task. 

2.  Participants  in  human-automation  interaction  studies  in  the  UAV  domain  arc  not  UAV 
operator  practitioners  but  undergraduate  and  graduate  students  |I4|[15](l6j|I7| 

[271(43). 

3.  Operators  arc  being  placed  in  both  single-task  [5]  [27)  and  multiple-task  [  14)[I5]  [  1 6) 

1 17|  [37)[43|  environments. 

4.  The  synthetic  task  may  involve  control  of  one  1 14)  [  15)  [  16)  1 17)  [43]  or  multiple 

[35)(37]  UAVs- 

5.  The  range  of  synthetic  tasks  is  broad  and  not  consistent  between  studies  at  one  institution, 
let  alone  between  institutions. 


6.  Types  of  automation  simulated  include  tracking,  route  planning,  system  monitoring,  and 
target  acquisition.  The  reliability  of  the  automation  was  the  independent  variable  in  most 
of  the  studies.  However,  in  no  study  is  information  about  automation  process  or  context 
presented  in  the  interface.  Moreover,  w  hile  automation  performance  information  is 
frequently  provided,  it  is  usually  introduced  either  as  part  of  the  training  or  as  feedback 
following  a  trial.  These  findings  were  very  surprising  to  us.  While  we  were  not  expecting 
to  find  research  reports  that  referred  explicitly  to  process,  performance  or  context 
information  in  an  interface,  we  did  expect  to  find  it  in  interfaces  for  UAV  automation. 
Although  it  is  possible  that  we  have  overlooked  relevant  literature,  it  appears  that  very 
little,  if  any.  research  has  been  reported  on  evaluations  of  interfaces  to  support  operator 
interaction  with  UAV  automation. 

Experimentation 

Cognitive  artifacts  were  designed  in  this  research  to  support  operator's  decision  making  process 
associated  with  automation  on  UA  Vs.  The  purpose  of  this  experiment  is  to  test  the  effect  of  those 
artifacts  on  operator's  interaction  with  automation.  In  the  experiment,  the  participants  will  be 
asked  to  complete  multiple  tasks  in  order  to  control  one  UAV  in  a  target  acquisition  mission. 
Sensor  fusion  automation  will  be  provided  to  help  them  detect  and  identify  hostile  targets.  The 
output  of  the  sensor  automation  contains  uncertainty  and  its  reliability  is  contingent  on  several 
context  factors  (c.g..  weather,  terrain  and  enemy  camouflage).  The  cognitive  artifacts  convey  the 
automation  process  and  the  context  information  to  operators.  We  expect  that  these  artifacts  will 
aid  participants  in  establishing  appropriate  levels  of  trust  in.  and  reliance  on.  the  automation.  We 
further  expect  an  overall  increase  of  human-automation  system  performance  in  the  target 
acquisition  mission. 

Introduction 

The  goal  (45)  to  enable  one  operator  to  control  multiple  UAVs  in  the  future  implies  that  more 
and  more  tasks  that  are  originally  assigned  to  human  operators  will  be  reallocated  to  automation. 
However,  automation  on  UAV'  systems  is  not  always  perfectly  reliable,  for  example,  the  result 
from  path  plan  automation  (20)  can  be  less  than  optimal,  and  the  sensor  fusion  automation  will 
be  affected  by  noisy  raw  data. 

Problematic  Use  of  Automation 

Human  factors  problems  of  imperfect  automation  have  been  noticed  and  studied  for  a  long  time. 
Parasuraman.  et  al.  (1997)  classified  these  problems  into  three  categories:  misuse,  disuse  and 
abuse  of  automation.  Misuse  occurs  when  operators  excessively  trust  the  automation  and  rely  on 
it  even  when  it  fails.  Studies  revealed  that  highly  reliable  automation  will  lull  operators  into  a 
state  of  complacency  (2)  (32)  which  undermines  operators'  ability  to  detect  and  respond  to 
automation  failures.  Disuse  happens  when  operators  mistrust  and  fail  to  rely  on  capable 
automation.  For  example.  Dzindolct.  et  al.  (2003)  found  that  people  distrust  even  reliable 
automation  after  observing  it  make  errors.  Whereas  the  first  two  problems  are  related  to 
operators,  abuse  of  automation  is  a  problem  of  system  designers  or  managers,  which  occurs 
when  they  incorporate  as  much  as  automation  into  the  system  without  considering  the 
consequence  on  human  operators,  such  as  mental  workload,  manual  skill  degradation,  loss  of 
situation  awareness  and  complacency  (33).  Parasuraman.  et  al.  (2000)  developed  the  model  of 
types  and  levels  of  automation  to  guide  the  implementation  of  automation  in  a  particular  system. 


Trust  in  Automation 

The  problematic  use  of  automation  occur*  when  people  fail  to  rely  on  the  automation  properly 
(24].  People's  reliance  on  automation  is  affected  by  many  factor*,  such  as  self  confidence,  trust 
in  automation,  perceived  risk,  and  fatigue,  amongst  others  (31].  Among  all  these  factors,  trust 
has  been  shown  to  be  particularly  important  for  understanding  human  reliance  on  automation. 
Therefore,  engendering  appropriate  trust  is  seen  as  critical  to  supporting  appropriate  reliance  on 
automation. 

The  appropriateness  of  trust  depends  on  the  user's  knowledge  about  the  automation  aid:  the 
performance  history  of  the  automation,  the  automation  process,  and  the  purpose  of  the 
automation  (24].  In  each  of  the  UAV  human-automation  studies  that  we  have  reviewed,  the 
purpose  of  automation  has  been  provided  to  operators  during  training.  In  many  studies,  an 
indicator  of  automation  performance  (or  expected  performance)  is  provided  as  well,  often  in 
terms  of  a  reliability  level.  Dzindolct.  ct  al.  (2003 )  suggested  that  it  was  not  realistic  to  provide 
continuous  feedback  of  automation  performance  to  operators.  Therefore  in  this  experiment,  only 
cumulative  feedback  of  the  automation  performance  w  ill  be  provided  to  participants  after  each 
trial.  With  regard  to  process.  Lee  &  See  (2004)  recommended  showing  the  algorithms  of  the 
automation  by  revealing  intermediate  results. 

For  example,  target  detection  automation  first  returns  real-valued  detection  scores  for  each 
suspicious  object  and  then  thresholds  these  scores  into  binary  categories  to  cue  operators  (38).  St. 
John,  ct  al.  (38)  tried  to  help  operators  make  better  use  of  unreliable  target  detection  automation 
by  revealing  the  real-valued  detection  scores  to  operators.  This  changed  the  target  detection 
automation  from  a  target  detector  to  an  information  tool  for  directing  search.  And  the  result 
indicated  a  general  improvement  in  participants*  detection  performance. 

One  concern  about  revealing  intermediate  results  to  operators  is  the  possible  increase  in 
workload  due  to  the  interpretation  of  those  results  (7). 

Context,  Automation,  and  Trust 

Lee  &  See  (2004)  demonstrate  that  the  appropriateness  of  trust  can  be  divided  into  three 
dimensions:  calibration,  resolution,  and  specificity.  Calibration  is  the  correspondence  between 
the  trust  level  and  the  true  quality  of  automation:  resolution  refers  to  how  precisely  trust 
differentiates  levels  of  automation  capability:  specificity  is  the  change  of  trust  for  different 
components  in  an  automation  system  or  over  time.  Calibration  emphasizes  the  value  —  the 
appropriate  level  of  trust,  while  resolution  and  specificity  highlight  the  discriminability. 
Sometimes  the  automation  performance  is  not  constant,  and  it  can  change  over  time  or  in 
different  situations  (7).  For  example,  the  performance  of  UAV  surveillance  automation  (20] 
varies  according  to  the  task  characteristics,  such  as  the  number  of  targets,  target  distribution, 
target  cost  variability  and  spatial  scale.  In  these  eases,  the  correct  calibration  of  trust  is 
unachievable  without  high  resolution  and  specificity,  therefore  resolution  takes  precedence  over 
calibration  (7]. 

One  important  reason  for  the  variation  of  automation  performance  is  that  there  are  contextual 
factors  in  the  environment  that  may  not  be  explicitly  considered  in  the  automation  algorithm  but 
still  affect  the  automation  performance  (24]  (28].  Cohen,  ct  al.  (7]  asserted  that  the  “user  must 
learn,  or  be  trained,  to  recognize  and  act  on  uncertain  quality  about  the  automation  and  to 
understand  how  such  uncertainty  can  change  from  situation  to  situation**  (pp.l ).  Their  dynamic 


model  of  trust  and  reliance  on  automation  (?]  indicates  that  the  context  can  influence  the 
performance  of  the  automation  and  therefore  the  calibration  of  trust  often  depends  on  how 
sensitive  people  are  to  the  influence  of  context.  The  design  implication  common  to  both  models 
is  that  automation  designers  must  help  users  to  relate  context  to  the  capability  of  the  automation. 

Several  studies  have  attempted  to  explore  the  influence  of  revealing  context  to  automation  usage: 

1 .  Dzindolct.  et  al.  (2003 )  asked  participants  to  indicate  the  presence  or  absence  of  a 
camouflaged  soldier  in  still  images  with  an  imperfect  automated  decision  aid.  In  their 
first  study,  they  found  that  participants  tend  to  distrust  reliable  aids  after  having  observed 
it  to  make  errors.  In  an  attempt  to  mitigate  this  effect,  participants  in  their  second  study 
were  provided  with  an  explanation  as  to  why  the  automated  aid  might  err:  the  decision 
aid  may  confuse  a  tree  with  soldier  if  the  tree  is  in  a  human-like  form.  The  results 
showed  that  the  explanation  increased  participants*  trust  and  reliance  in  decision  aid.  For 
those  participants  who  were  using  highly  reliable  automation,  the  increase  of  trust  and 
reliance  was  appropriate.  However,  for  those  using  unreliable  automation,  the  increase  in 
trust  was  inappropriate. 

2.  Parasuraman.  ct  al.  (1993)  examined  people’s  reliance  on  system  monitoring  automation 
in  a  multi-task  environment,  in  which  automation  with  constant  reliability  lulled  the 
participant  into  complacency.  Bagheri  (2004)  replicated  this  paradigm  but  informed  the 
participants  the  context-related  nature  of  automation  reliability.  The  result  revealed  that 
providing  context  information  mitigated  the  complacency  effect  and  improved 
participants’  performance  with  constant  and  highly  reliable  automation. 

3.  Masalonis  &  Parasuraman  (  2003)  tested  participants*  use  of  air  traffic  control  decision 
aids.  Two  groups  of  participants  were  asked  to  detect  potential  conflicts  between 
approaching  aircraft  pairs  with  the  aid  of  an  imperfect  conflict  detection  algorithm.  The 
reliability  of  the  automation  varied  corresponding  to  two  different  scenarios,  and  this 
information  was  only  given  to  one  of  the  two  groups  through  training.  The  result  showed 
that  both  trust  and  reliance  were  affected  by  this  context  information.  The  trained  group 
trusted  automation  in  the  two  scenarios  differentially  and  more  appropriately.  However, 
their  overall  performance  didn’t  improve.  This  was  because  that  the  context  information 
appeared  to  have  a  side  eff  ect  of  reducing  participants*  criteria  to  report  a  conflict, 
whereas  the  trained  group  tended  to  accept  the  automation  judgment  without  question. 

These  studies  provide  initial  evidence  that  providing  operators  with  information  related  to  the 
context  affecting  automation  reliability  docs  influence  their  trust  and  reliance  on  that  automation. 
This  knowledge  appears  to  have  helped  participants  to  more  effectively  allocate  attention  to  the 
automated  task  |2|  and  benefited  the  resolution  and  specificity  of  trust  across  situations.  Thus. 
Lee  and  Sec’s  (2004)  recommendation  to  “reveal  the  context  and  support  the  assessment  and 
classification  of  the  situations  relative  to  the  capability  of  automation”  (pp.  75)  appears  to  have 
empirical  support. 

However,  contextual  information  may  also  have  a  down  side,  inducing  some  participants  to  rely 
on  the  automation  even  when  trust  is  unwarranted  (IS)  (28].  Therefore,  the  recommendation  to 
reveal  context  information  must  be  tempered  by  the  recognition  that  this  information  may  have 
an  undesirable  effect  on  an  individual’s  decision  criterion.  Striking  a  balance  between  these 
concerns  will  require  more  extensive  empirical  evidence. 


In  all  three  of  the  studies  mentioned  above,  context  was  revealed  through  training.  Although  this 
approach  offers  a  relatively  easy  way  to  study  the  role  of  context  in  operator  trust  and  reliance, 
there  are  several  drawbacks  to  training.  An  alternative  approach  to  communicating  contextual 
factors  is  to  include  them  in  the  operator  interface. 

We  propose  to  design  cognitive  artifacts  that  convey  the  contextual-dependence  of  automation 
behavior.  The  interface  approach  may  be  preferable  to  the  training  approach  for  two  reasons. 
First,  training  requires  operators  to  remember  the  context  information  and  recall  it  later  during 
mission,  and  this  occupies  resource  of  working  memory  (42).  In  stressful  situations,  working 
memory  capacity  is  reduced  (and  the  consequences  of  failure  increase).  An  interface  could  places 
knowledge  contextual  impact  in  the  world,  thereby  relieving  the  operator  of  a  memory  burden. 
Second,  the  nature  of  training  determines  that  it  can  only  provide  information  for  anticipated 
situations.  However,  an  interlace  can  be  designed  in  a  way  to  provide  useful  information  even  in 
unanticipated  situations,  which  characterize  complex  UAV  missions  (4). 

Methods 

The  methods  to  be  used  in  this  experiment  are  drawn,  in  pari,  from  those  described  by  Wickens. 
et  al.  (2005).  This  is  to  take  advantage  of  their  prior  experience  in  conducting  human-automation 
interaction  studies  in  a  similar  STE.  Taking  this  approach  will  reduce  risks  in  our  experimental 
design  without  compromising  our  research  objectives.  As  well,  it  will  be  advantageous  to  be  able 
to  compare  results  with  those  obtained  by  Wickens.  et  al.  (2005) 

Participants 

Graduate  students  at  University  of  Toronto  Institute  for  Aerospace  Studies  (IJTIAS)  will  be 
recruited  to  participate  in  this  experiment.  Graduate  students  were  used  successfully  by  Wickens, 
et  al.  (2005)  and  UTIAS  students  have  participated  in  research  studies  in  the  past.  These  students 
arc  also  the  most  convenient  participants  for  this  experiment  because  the  STE  will  be  located  at 
UTIAS,  which  is  removed  from  the  main  campus  of  the  University  of  Toronto  .  The  participants 
will  receive  cash  compensation  and  a  bonus  will  be  given  to  top  performers.  The  compensation 
scheme  follows  that  used  by  Wickens.  et  al.  (2005)  and  has  been  found  to  be  adequately 
motivating. 

Synthetic  Task  Environment 

Vega  Prime  2.0  Simulator 

Vega  Prime  is  judged  to  be  a  leading  candidate  for  a  synthetic  task  environment  for  human 
factors  UAV  research' .  Vega  Prime  is  a  commercial-off-the-shelf  software  application  for  the 
creation  and  deployment  of  real-time  313  simulation,  training,  and  visualization  applications.  It  is 
a  high-fidelity  environment,  allowing  for  realistic,  interactive  acnal  simulations.  Its  Graphical 
User  Interface  (GUI),  Application  Program  Interface  (API),  and  many  extensible  modules 
provide  a  high  degree  of  flexibility  and  functionality.  Vega  Prime  is  used  at  several  universities 
in  North  America,  including  the  University  of  Toronto  Institute  for  Aviation  Studies  (UTIAS). 


...bul  close  to  DRD( -Toronto 

*  An  early  list  of  UAV  simulators  is  included  in  Appendix  A  fc c  information.  The  list  is  incomplete  and  does  not  reflect  *on»c  of 
the  simulate**  described  above. 


The  information  below  was  compiled  from  communication  with  UT1AS  staff.  Vega  Prime 
vendors,  university  developers,  the  Vega  Prime  Programmer's  Guide,  a  visit  to  the  UT1AS 
facility,  and  content  from  the  Vega  Prime  website. 

Vega  Prime  Knvironment  at  UTIAS 

UTI  AS  currently  operates  a  Vega  Prime  environment  consisting  of  three  software  components: 

•  Vega  Prime  is  the  scene  graph  manager  that  functions  as  the  foundation  of  the  program. 

It  contains  core  C++  classes  and  functions.  A  screenshot  of  a  sample  scene  window  is 
shown  in  Figure  15. 

•  LynX  Prime  is  a  GUI  tool  that  allows  users  to  configure  the  scene  graph  in  Vega  Prime. 
It  also  contains  an  API  to  support  programming  interfaces  in  Vega  Prime  (generally  for 
more  experienced  users).  This  is  shown  in  Figure  16. 

•  GL  Studio  is  a  tool  that  aids  creation  of  2D  or  3D  graphical  displays,  including 
instrumentation  models  for  simulation  or  training  applications.  GL  Studio  Plug-In  allows 
the  graphics  to  be  integrated  into  Vega  Prime  such  that  the  instrumentation  can  interact 
with  the  scene  graph.  For  example,  a  dial  graphic  can  be  easily  linked  to  the  pitch  of  a 
plane  without  the  need  to  write  additional  code.  An  example  of  the  instrumentation 
generated  is  shown  in  Figure  17. 

The  following  Vega  Prime  modules  arc  not  currently  available  at  UTIAS  but  may  be  of  value  to 
UAV  human-automation  interaction  experiments9: 

•  Vega  Prime  IR  Scene  provides  real-time,  physics-based  generation  of  a  scene  at  a  variety 
of  wavelengths  from  visible  to  infrared.  It  achieves  this  primarily  using  a  physics-based 
approach  to  calculate  the  apparent  radiance  of  objects  w  ith  respect  to  the  viewpoint  of 
the  user.  A  sample  screenshot  is  shown  in  Figure  IS. 

■  Vega  Prime  IR  Sensor  is  an  addition  to  IR  Scene  that  adds  realistic  sensor  effects,  such 
as  blur,  saturation,  jitter  and  various  types  of  noise,  to  scenes  generated  using  IR  Scene. 
Post-processing  done  by  IR  Sensor  helps  to  simulate  sensor  devices  such  as  night-vision 
goggles  or  long-range  infrared  systems.  A  sample  screenshot  is  shown  in  Figure  IS 

•  Vega  Prime  Radar  provides  real-time,  realistic  3D  radar  displays.  A  sample  screenshot  is 
shown  in  Figure  19. 

•  Vega  Pnme  MAT  creates  atmospheric  conditions. 

•  Vega  Pnme  TMM  (Texture  Matenal  Mapper)  makes  it  possible  to  assign  material 
classifications  to  textures  in  an  IR  database,  w  hich  adds  detail  and  realism  to  simulated 
sensor  scenes.  A  sample  database  is  included.  This  is  being  rolled  into  a  module  called 
"CTS  Sensors.** 


Pricing  intnmuikm  forth:**;  modules  is  provided  in  Apprixiix  B. 


Hgurr  16:  Scrrvmliul  of  L>n\  Prime  application 


Figure  18:  Screciuliot  <>!  win*  crated  u*ing  Vega  Prime  IK  Scene  (lefl)  ami  \  ega  Prime  IK  Sen  nor  f  ri|*ltl  > 


Figure  19:  ScreenUiot  of  a  Vega  Prime  %cene  (lefl)  ami  currapondint:  radar  return  created  tiring  Vega  Prime 
Kadar  (right) 


UTIAS  runs  Vega  Prime  an  a  single  Windows-based  system  with  multiple  monitors,  including  a 
touch  screen.  Vega  Prime  is  available  in  two  areas:  one  office  space  and  one  large  laboratory  that 
contains  a  motion-based  llight  simulator.  Additional  computers  and  monitors  arc  present  in  both 
areas.  Some  rearrangement  of  the  monitor*  would  likely  be  required  to  create  a  functional  testing 
environment,  but  there  is  sufficient  room  and  flexibility  in  the  hardware  systems  to  do  so. 


Implementation  Considerations 

Vega  Prime  is  a  leading  candidate  for  an  STE  for  UA V  human-automation  interaction.  It  has  the 
following  advantages: 

•  The  progress  of  the  UAV  can  be  simulated  as  if  the  aircraft  were  flying  between 
waypoints  and.  if  necessary,  loitering  around  a  fixed  point. 

•  The  sensor  modules  (i.e..  optical  camera.  IR.  radar,  etc.)  allow  for  simulation  of  the 
scene  information  obtained  by  UAV  sensors.  For  example,  we  can  simulate  the  image 
obtained  from  a  camera  mounted  on  the  UAV  panning  across  the  scene.  This 
functionality  is  already  partially  implemented  in  the  IJTIAS  configuration. 

•  The  display  of  the  scene  graph  can  be  manipulated  with  the  Vega  Prime  API  to  cue 
objects  in  the  scene.  This  requires  only  a  moderate  amount  of  coding. 

•  Automation  can  be  simulated  with  a  “Wizard  of  Oz"  technique — by  scripting  where, 
when,  and  how  automation  tasks  arc  performed.  This  will  allow  for  greater  experimental 
control. 

■  Instrumentation  can  he  easily  linked  to  attributes  in  the  scene  through  GL  Studio. 

A  potential  drawback  of  the  Vega  Prime  system  (at  least  in  the  UTIAS  implementation)  is  that  it 
does  not  have  flight  dynamics  module  or  an  image  database  for  a  UAV'.  Thus,  they  arc  currently 
unable  to  simulate  the  flight  characteristics  of  a  UAV'  and  they  cannot  generate  an  image  of  a 
UAV.  These  drawbacks  may  not  be  serious10.  Given  that  we  do  not  intend  to  include  the  piloting 
task  in  our  experiment,  the  flight  dynamics  will  not  come  in  to  play.  UTIAS  has  many  aircraft 
models  and  it  would  be  possible  to  select  one  that  has  a  similar  flight  envelope  to  a  UAV, 
yielding  a  sufficient  level  of  fidelity  for  the  experiment.  The  inability  to  view  the  UAV'  itself  is 
also  not  a  substantial  problem  because  there  is  no  need  to  see  a  UAV  in  the  scene.  Rather  the 
tasks  involve  viewing  the  scene  from  an  egocentric  perspective. 

Experimental  Design 

A  2x2  bctwccn-subjccts  repeated  measures  design  will  be  employed  in  the  experiment  to  test  the 
effect  of  two  types  of  information  contained  in  the  interface:  I )  automation  process  information 
(present  or  absent),  and  2)  context  information  (present  or  absent).  Thus,  four  experimental 
groups  will  be  formed: 

•  Control  Group:  process  information  absent  context  information  absent 

•  Treatment  Group  1:  process  information  present  context  information  absent 

•  Treatment  Group  2:  process  information  absent  context  information  present 

■  Treatment  Group  3:  process  information  present  context  information  present 

Experimental  Tasks 

Each  participant  will  complete  10  mission  trials  (scenarios),  each  lasting  approximately  10 
minutes.  In  each  trial,  the  UAV  will  fly  a  pre-determined  path  through  a  scene  populated  with 
targets.  The  primary  task  will  be  to  identify  the  targets  in  the  scene.  Participants  will  be  aided  in 


*•  We  have  not  enquired  with  the  UIUC  about  w  hether  they  arc  u*ing  L  AV  flight  and  image  models.  This  might  offer  anther 
way  lo  circumvent  ihit  drawback. 


this  task  by  target  identification  automation  comprised  of  two  ATR  systems  and  the  DS  sensor 
fusion  automation  described  above  (see  Figure  20). 


Target  Identification  Automation 


Influenced  by  Context  Automation  Process 

Revealed  in  Interface 


Figure  20.  Senior  fusion  automation  (Ion  chart 

It  may  be  necessary  to  introduce  secondary  tasks  to  increase  the  workload  level1 '.  An  effort  will 
be  made  to  keep  the  difficulty  level  of  each  mission  consistent.  However,  the  order  of 
presentation  of  the  scenarios  will  be  randomized  to  minimize  order  effects. 

The  location  of  hostile  targets  detected  by  the  sensor  fusion  automation  will  be  cued  on  the 
tactical  situation  map  and  the  type  of  target  will  be  indicated  with  text.  In  the  three  treatment 
conditions,  additional  information  about  the  automation  process  and/or  the  context  factors  will 
be  revealed  through  the  interface  as  described  above.  The  participant  will  respond  by  pressing  a 
key  or  touchscreen  location  to  designate  the  type  of  target.  The  participant  will  be  free  to  use  the 
automation  process  or  context  information  (where  provided)  or  the  EO/1R  camera  (all  groups)  to 
acquire  information  about  the  automation  conclusion  or  the  visual  scene,  respectively.  The 
training  of  the  EO/IR  camera  will  be  automated  in  the  simulation  to  control  for  spatial 
orientation  skills.  A  key  challenge  lies  in  tuning  the  experimental  scenario  by  balancing  the 
payoff  function  and  imposing  time  constraints  (including  delays  in  the  camera  pan  and  zoom)  on 
task  completion. 

This  automation  will  not  make  target  detection  errors.  That  is  to  say  that  all  objects  cued  by  the 
automation  will  be  targets.  However,  the  target  identification  could  be  wrong  in  either  one  of  the 
ATR  conclusions  or  in  the  DS  sensor  fusion  conclusion.  The  automation  reliability  is  computed 
as  the  rate  of  correct  target  type  identification  by  the  DS  sensor  fusion  automation.  The 
automation  reliability  will  be  constant  within  each  trail,  but  will  vary  among  trails  with  different 
context  In  the  10  trails,  the  sequence  of  the  automation  reliability  will  be  balanced  across 
participants. 


11  For  c.vimptc.  Wicked*  d  at.  (38)  introduced  a  ’’target  ol*  opportunity"  detection  task  and  a  system  monitoring  task  »o  ircrease 
mxvfckud 


Participants  will  be  given  feedback  about  the  performance  of  each  type  of  automation  (i.c.,  ATR 
and  sensor  fusion)  at  the  completion  of  each  trial.  Note  that  this  information  is  prov  ided  for  all 
groups. 


Procedure 

A  detailed  experimental  procedure  will  not  be  presented  here.  The  procedure  will  generally 
consist  of  4  phases:  Informed  consent,  collection  of  demographic  measures,  experimental 
scenarios,  and  debriefing.  Breaks  will  be  given  breaks  between  the  phases  and  at  regular 
intervals  between  the  trials.  The  total  experiment  is  expected  to  last  2.5-3  hours. 

Measures  and  Instruments 

The  measures  to  be  collected  in  the  experiment  are  listing  in  Table  2  along  with  the  instruments 
that  will  be  used  to  collect  them. 


Table  2.  Experimental  measures  and  associated  instruments. 


Occurrence 

Type  of 
Measure 

Data  C  ollected 

Instrument 

Phase  2:  One 

1 1  mr* 

Demographic 

General  predisposition  to 
trust 

four  questions  about  trust  in 
everyday  objects  (26). 

ii ii iv 

General  self-confidence 

Four  questions  about  self- 
confidence  in  everyday  tasks 
(261. 

Phase  3: 

During  each 
trial 

Objective 

Participant  identification  of 
each  target 

Simulation  software 

Identification  time  for  each 
target 

Simulation  software 

Verification  strategy  for 
each  target  ~ 

Simulation  software 

Performance  of  secondary 
tasks  (if  included) 

Simulation  software 

Phase  3: 

After  each 
trial 

Subjective 

Perceived  workload 

NASA  Task  Load  Index  (TLX) 

Self-confidence  to  fulfill 
target  identification  with 
automation 

One  direct  question  ( 10-point 
scale)  developed  by  Lee  & 

Moray  (1992, 1994) 

Tmst  in  the  automation 

Two  options: 

One  direct  question  (10-point 
scale)  developed  by  Lee  & 

Moray  (1992,  1994). 

The  1 2-item  trust  scale 
designed  by  Jian.  ct  al.  (2000). 

Note  that  the  operator's  self-confidence  in  her  ability  to  complete  tasks  manually  and  her  trust  in 
automation  have  long  been  used  as  subjective  measures  in  trust  in  automation  research.  However, 
Masalonis.  ct  al.  (28 J  suggested  that  as  users  learned  how  to  accommodate  the  automation's 
failings  and  capitalize  on  its  strong  points,  it  will  be  more  meaningful  to  questions  an  operator's 
self-confidence  in  his  ability  to  complete  the  tasks  with  the  automation.  We  include  all  three 
measures 


12  Whether  or  nx.  parlicipu**  direct  the  video  camera  to  the  target  location. 


Hypotheses: 


Main  Effects 

H 1 .  Participants  receiving  automation  process  information  will  exhibit  trust  that  is  better 
calibrated,  more  resolved,  and  more  specific  as  compared  to  participants  not  receiving 
this  information. 

H2.  Participants  receiving  automation  process  information  will  perform  better  on  the  target 
identification  task  as  compared  to  participants  not  receiving  this  information. 

H3.  Participants  receiving  automation  context  information  will  exhibit  more  better 

calibrated,  more  resolved,  and  more  specific  as  compared  to  participants  not  receiving 
this  information. 

H4.  Participants  receiving  automation  context  information  will  perform  better  on  the  target 
identification  task  as  compared  to  participants  not  receiving  this  information. 

Test  and  Evaluation  Plan 

The  purpose  of  this  section  is  to  outline  a  test  and  evaluation  plan  for  the  experiment  described 
above.  Developing  a  synthetic  task  environment  and  experimental  scenarios,  implementing 
human-machine  interfaces,  and  linking  the  various  elements  together  are  high  risk  tasks  that  will 
be  prone  to  disruptions.  Therefore,  the  tasks  in  this  plan  should  be  considered  to  be  the  minimum 
required  to  undertake  the  experiment.  Similarly,  the  timeline  provided  is  only  an  approximation 
and  should  be  re-evaluated  on  a  regular  basis  as  the  plan  is  executed. 


Task 

Milestone 

Duration 

Completion 

Ketine  expet ltncnlal 
pbn'5 

DRDC  sign -off  on 
experimental  pbn 

1  month 

May  31*.  2006 

Acquire  raiiir  and  1R 
module*  for  VEflA 

Prime 

1  month 

Develop  Synthetic  la *k 

environment 

IX'tiuinvtiiiikm  ol 
luik-tionil  capabiluies  of 
STE 

2  months 

Aug.  31  . 21X16 

Implement  interface 
artifact* 

Delivery  of  software 
code 

1  month 

Develop  experimental 

scenarios 

Demonstration  ol  ten 
experimental  scenarios 
riveting  the 
requirements  of  the 
expen  menial  pbn. 

1  month 

Oct.  31  , 2006 

Collccl  and  ptvpare 
instruments  and 

Provide  completed 
checklist  of  measures 

1/2  month 

Nov.  15".  2006 

11  Refinement  tasks  will  sk  hide:  designing  ihc  appearance  of  ih:  targeting  cue.  determining  a  compensation  plan,  setting  th: 
initial  weights  of  the  pa\xitY  function,  setting  the  initial  levels  of  automatics  reliability. 


documcnlatinn14 

and  associated 

itniiumenl?* 

Conduct  u  usability- 
study  im  the  interlace 
aina'pU 

1/2  month 

Nuv.  3tf\  2006 

Kun  piiui  study 

1  nxmtli 

Dev.  31*.  2006 

Sumnntiv  e  review  of 

expen  menial  dextgn 

Verbal  agreement 

between  DRIX'  and 
contractor  that 
experiment  is  fully 
prepared 

2  ntinlhs 

leh  3tf\  2UD6 

IVOdlHtt  lift!  ol 

participant  and  ihiar 
itsiimmcnl  lu  giuups 

1  nxinlli 

March.  31"  2IMX. 

Data  ndlemon 

All  participants 

completed 

1  nxinlli 

April.  3(1".  2IXW. 

Data  recovery  and 

treatment 

Confirmation  that  all 

data  hav  e  been  recorded 
and  extracted  as 
required  for  analysis 

1  nxinlli 

May.  31 ‘.2006 

Data  Analyst* 

Produce  complete 
statistical  analyses  uf  all 
measures 

1  month 

Jun  30“.  2006 

Reporting 

Final  report  delivered  in 

DR  DC 

AugusL  31".  2006 

Conclusions 

This  report  has  described  a  research  plan  for  testing  human-automation  interfaces  for  UAV 
automation.  The  experiment  is  grounded  in  theory*  and  practice  of  human  factors  scientists 
exploring  the  rule  of  trust  in  automation  as  an  important  determinant  of  human  automation 
reliance.  A  review  of  relevant  literature  in  the  domain  suggests  that  the  proposed  study  poses 
unique  and  important  questions  that,  if  answ  ered,  would  comprise  a  significant  contribution  to 
the  knowledge  base  supporting  UAV  automation  design.  Moreover,  the  results  w  ould  be  of 
broader  interest  to  the  human  factors  community  as  it  addresses  issues  that  cross  domain 
boundaries.  The  risks  inherent  in  the  undertaking  of  a  new  research  program  should  be  further 
resolved  and  weighed  against  the  expected  returns. 
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Appendix  B:  Pricing  for  Vega  Prime  Modules 

Tabic  3  lists  the  educational  pricing  for  Vega  Prime  modules  not  yet  available  at  IJTIAS.  Each 
additional  monitor  that  a  single  Vega  Pnmc  system  runs  on  requires  a  runtime  license.  UTIAS 
has  the  correct  classifications  and  Export  License  to  purchase  these  modules. 
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!  U)  This  article  presents  an  experimental  plan  tor  the  evaluation  ot  theory -driven 
human-automation  interface  concepts  in  the  domain  of  Unmanned  Aerial  Vehicles 
(UAVs).  The  experiment  is  motivated  by  the  evolution  of  UAV  technology  towards  more 
highly  automated  systems  and  the  relegation  of  human  operators  to  the  rote  of  supervisory 
controller.  Humans  have  consistently  demonstrated  themselves  to  be  ill-suited  for  this 
role:  thus,  v/e  anticipate  that  future  UAV  systems  will  be  susceptible  to  human-automation 
breakdowns  leading  to  mission  failures. 

In  an  allempt  to  head  off  such  failures,  human  factors  researchers  are  explonng  the  role  of 
human  frusf  in  automation  as  a  means  of  managing  the  uncertainty  inherent  in 
mixed-initiative  systems.  A  recent  comprehensive  review  of  fhe  trust  literature  yields 
design  guidelines  for  engendering  appropriate  levels  of  trust  in  automation.  We  followed 
these  guidelines  to  produce  example  interlace  artifacts  that  show  information  about  the 
automation  process  (i.e..  the  automation  algorithm)  and  relevant  context  information  that 
affects  the  automation's  behaviour. 

We  propose  an  outline  for  a  simulator  study  to  evaluate  these  interface  concepls  with 
human  operators.  The  study  casts  UAV  operators  opposite  sensor  fusion  automation  that 
identifies  largets  based  on  analysis  of  multiple  sensor  images.  The  task  of  the  joint 
human-automation  system  is  to  correctly  identify  friendly  and  hostile  military  targets  in  a 
scene.  The  UAV  operators  will  be  assigned  to  four  groups  to  test  the  differential  effects  of 
providing  information  about  automation  context  and  process.  We  hypothesize  that 
providing  either  or  both  of  these  types  of  information  about  automation  will  engender  more 
appropriate  trust  in  the  automation  and  lead  to  performance  improvements  over  a  control 
condition  lacking  this  information.  The  results  would  constitute  a  unique  and  substantial 
contribution  to  both  the  human-automation  interaction  literature  and  to  the  ongoing 
development  of  UAV  automation. 

!U)  Les  vdhicules  adriens  sans  pilote  (UAV)  sont  rapidement  en  tram  de  devenir  une  partie 
mtdgrante  des  operations  militaires  contemporaines.  A  la  lumidre  des  progrds  rdalisds 
dans  les  technologies  de  vol  et  de  charge  utile  des  UAV.  le  rdle  des  opdrateurs  d'UAV  est 
en  tram  de  se  transformer  en  celui  de  contrdleur-superviseur  de  sysffime  automated 
complexe.  Cependant.  il  est  bien  dtabli  que  les  opdrateurs  humains  sont  mal  adaptds  pour 
ce  rdle.  Une  masse  croissante  de  documentation  tdmoigne  du  rdle  crucial  que  joue  la 
confiance  dans  I'automatisation  pour  determiner  I'efficacitd  du  contrdle  humain  sur  des 
systdmes  automatisds.  D'aprds  une  rdcente  compilation  de  la  recherche  mende  sur  la 
confiance  dans  I’automatisation.  il  serait  essentiel  de  procurer  aux  op6rateurs  des 
informations  sur  le  processus  d'automatisation  et  sur  la  fiatolitd  de  I'automatisation  en 
contexte  spdcilique  pour  promouvoir  un  degrd  appropnd  de  confiance  dans 
I'automatisation. 

Le  prdsent  rapport  prdsente  un  plan  de  recherche  visant  A  crder  des  conceptions 
novatrices  pour  les  artefacts  cognitifs  communiquant  ces  deux  types  d  informations  aux 
opdrateurs  d'UAV  dans  le  but  d'engendrer  la  confiance  appropride  dans  les  systdmes 
automatisds.  Cela  inclut  une  categorisation  des  nouvelles  technologies  d'automatisation 
dmergeant  dans  les  systdmes  d'UAV.  une  analyse  des  tdches  changeantes  des 
dquipages  d'UAV.  et  un  examen  sdlectif  de  la  recherche  pertinents 
homme-automatisation.  Le  rapport  introduit  ensuite  deux  concepts  d'interface 
homme-machme  pour  des  types  sdlectionnds  d  automatisation  d'UAV  et  il  propose  un 
plan  de  recherches  pour  dprouver  et  dvaluer  ces  conceptions. 


